High Performance RDMA Protocols in HPC

Modern network interconnects that leverage Remote Directory Memory Access (RDMA) and OS bypass, such as Infiniband [2], Myrinet [9], and iWARP over TCP [3], can offer significant performance advantages over conventional send/receive network semantics. However, the high performance of RDMA often comes with hidden costs. RDMA based interconnects generally fail to provide true one-sided semantics, requiring an exchange of information prior to initiating a one-sided RDMA operation. In addition, both the initiator and target must typically preserve the physical to virtual memory mappings during the RDMA operation. This paper describes a unique user-level ‘pipeline’ protocol that addresses these constraints while avoiding some of the pitfalls of existing techniques. By effectively overlapping the cost of memory registration with RDMA operations this protocol provides good performance even in the absence of memory buffer reuse. This protocol may also take advantage of memory buffers that have already been used in RDMA operations by avoiding the cost of memory registration. Through this approach, bandwidth may be increased up to 67% when memory buffers are not effectively reused while providing performance equal to that of existing techniques as demonstrated by both Linpack and NPB benchmark results. Several user level protocols are explored using Open MPI’s PML (Point to point messaging layer) and compared/contrasted to this ‘pipeline’ protocol.